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Abstract 

Distributed  simulation  provides  warfighters  with  training  to  enhance  their  team  and  inter-team  skills  with  greater 
frequency  and  at  lower  cost  than  range  training  exercises.  Distributed  simulation  training  for  small  groups  of 
warfighters  such  as  a  formation  of  four  fighters  working  with  an  Air  Battle  Manager  can  be  focused  on  specific 
skills  such  as  beyond-visual  range,  dissimilar  air  combat  tactics  using  constructive  simulations  as  adversary  forces. 
Training  for  command  and  control  teams,  however,  requires  interactions  among  blue  force  entities,  particularly 
voice  communication,  that  cannot  be  supported  using  only  constructive  simulations.  One  solution  is  to  conduct 
large  scale  virtual  training  events  such  as  VIRTUAL  FLAG  exercises.  Another  solution  is  to  combine  human-in- 
the-loop  virtual  simulators  with  white-force  role-players  who  provide  responsive  verbal  communications  for 
constructive  entities.  Both  of  these  approaches  require  participation  from  a  significant  number  of  warfighters  or 
subject  matter  experts  which  increases  training  cost  and  decreases  ability  to  focus  training  on  command  and  control 
teams  such  as  Air  Battle  Managers.  To  overcome  these  difficulties,  Australian  and  US  researchers  conducted 
Exercise  Pacific  Link  2  in  which  an  Air  Battle  Manager  team  in  Melbourne,  Australia  was  networked  with  a  four- 
ship  of  F-16  simulators  and  a  constructive  forces  simulator  in  Mesa,  Arizona.  Using  a  novel  approach  to  scenario 
design  combined  with  an  improved  constructive  entity  generator,  a  small  team  of  pilots  and  engineers  provided  five, 
fully  interactive  four-aircraft  formations  of  F-  16s  which  engaged  multiple  waves  of  adversary  aircraft  over  a  one- 
hour  vulnerability  period.  Evaluations  from  the  Air  Battle  Management  team  demonstrate  significant  training 
benefits  from  this  approach.  Data  will  be  presented  on  constructive  forces  improvements  and  results,  system  and 
scenario  design  together  with  feedback  from  exercise  participants  regarding  skills  that  were  enhanced  and 
opportunities  for  further  developments. 
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Air  Battle  Managers  (ABMs),  ground-based  or 
airborne,  work  as  a  team  to  coordinate  and  manage 
forces  engaged  in  air  operations.  Simulator  training  for 
ABMs  requires  interaction  with  a  complex,  responsive, 
composite  force  and  an  unpredictable  adversary. 
Researchers  in  Australia  have  gained  considerable 
experience  in  developing  training  systems  for  ABMs 
while  US  researchers  have  focused  their  efforts  on 
training  for  fighter  pilots.  The  Air  Operations  Division 
of  Australia’s  Defence  Science  and  Technology 
Organisation  (DSTO/AOD),  Melbourne,  Victoria  and 
the  US  Air  Force  Research  Laboratory’s  Warfighter 
Readiness  Research  Division  (AFRL/HEA),  Mesa, 
Arizona  are  conducting  research  efforts  to  enhance  the 
scope  and  capabilities  of  warfighter  training  using 
distributed  simulation.  In  2005,  DSTO/AOD  and 
AFRL/HEA  conducted  Exercise  Pacific  Link,  a 
coalition  mission  training  event  designed  to  assess  the 
ability  of  an  Internet  Protocol  (IP)-based  distributed 
simulation  network  to  support  Distributed  Mission 
Training  (DMT)  for  air  combat  (Crane  et  al,  2006). 
During  Exercise  Pacific  Link  (PL1),  a  series  of  training 
scenarios  of  increasing  complexity  was  carried  out  over 
three  days,  linking  fighter  pilots  and  ABMs  at  both 
sites  with  distributed  mission  briefings  and  debriefs.  In 
2006,  DSTO/AOD  and  AFRL/HEA  conducted 
Exercise  Pacific  Link  2  (PL2)  with  the  aim  of  using 
lessons  learned  from  the  first  Pacific  Link  exercise  to 
develop  new  systems  for  training  ABMs  using 
distributed  simulation. 

PACIFIC  LINK  2  OBJECTIVES 

Based  on  lessons  learned  from  the  scenario  limitations 
of  PL  1,  a  primary  objective  of  PL2  was  to  develop  and 
evaluate  systems  which  would  allow  a  small  White 
force  to  inject  multiple  entities  into  the  battlespace 
including  radio  communications,  weapons 
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engagements,  and  interactions  with  ABMs.  The  value 
of  DMT  for  enhancing  the  skills  of  ABMs  as  well  as 
pilots  has  been  described  since  the  earliest  research  on 
distributed  simulation  (Bell  &  Crane,  1993).  One 
constraint  on  the  quality  of  ABM  training  is  that  with  a 
small  number  of  virtual  fighter  events,  training  is 
limited  to  close-control  of  the  fighters  tactically 
employing  against  enemy  fighters.  While  this  is 
valuable  training  that  effectively  complements  range 
training,  other  aspects  of  battle  management  are  not 
trained.  These  tasks  include  force  management  with 
multiple  flights  of  fighters,  management  of  aerial 
refueling  assets,  and  contingency  operations.  More 
friendly  and  adversary  forces  than  can  typically  be 
provided  via  virtual  simulation  at  a  single  mission 
training  center  will  be  required  to  support  these  aspects 
of  ABM  training.  Coalition  force  DMT  could  be  an 
effective  system  for  providing  this  training.  In  live 
training,  one  group  of  warfighters  will  frequently 
provide  training  support  for  another  group  by  serving 
as  adversaries.  These  Red  Air  forces  represent  a 
significant  use  of  resources  with  little  benefit  for  the 
participants.  An  advantage  of  DMT  is  that  these 
dedicated  training  forces  can  often  be  replaced  with 
constructive  (computer-generated)  entities  which 
reduces  training  costs  but  also  reduces  interactivity, 
scenario  flexibility  and  responsiveness  of  simulated 
entities.  One  solution  is  to  integrate  constructive 
entities  with  voice  actors  who  role  play  the  part  of 
friendly  force  elements  (Crane,  Tomlinson,  &  Bell, 
2002).  While  this  is  a  useful  approach,  it  requires  that 
each  voice  actor  be  a  subject  matter  expert  who  can 
communicate  correctly  in  accordance  with  standards 
and  execute  tactical  commands  as  required.  It  also 
requires  "user-friendly"  instructor-operator  interfaces 
easily  controlling  responsive  and  realistic  constructive 
players.  To  provide  training  for  a  team  of  ABMs, 
several  knowledgeable  voice  actors  would  be  required. 
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An  alternative  approach  was  employed  in  PL2 
combining  novel  scenario  design  with  a  new  approach 
to  scenario  implementation  using  constructive  forces 
which  could  fly  autonomously  or  be  controlled  by  a 
human  operator. 

Other  key  objectives  of  PL2  included:  continuing  the 
development  of  simulation  infrastructure  for 
conducting  Australia-US  DMT  exercises,  continuing 
the  development  of  team  performance  measurement 
and  feedback  techniques  DMT,  and  developing  and 
assessing  a  structured  method  for  designing  ABM  team 
training  scenarios.  This  was  done  by  drawing  on 
guidance  from  previous  research  into  DMT 
applications  and  team  training  including  the  USAF 
Mission  Essential  Competencies  (MECs;  e.g., 
Colegrove  &  Alliger,  2002),  the  Event-Based 
Approach  to  Training  (EBAT;  Fowlkes,  Dwyer,  Oser, 
and  Salas,  1998),  and  Team  Dimensional  Training 
(TDT;  e.g.,  Smith- Jentsch,  Zeisig,  Acton,  & 
McPherson,  1998)  as  well  as  by  linking  into  an 
Australian  task  framework,  the  Australian  Joint 
Essential  Task  list  (AS JETS;  McCarthy,  Kingston, 
Johns,  Gori,  Main,  &  Kruzins,  2003).  Measurement  of 
training  effectiveness  allowed  evaluations  of  the 
scenario  tools  and  constructive  forces  role-play 
supporting  these  mission  areas,  tasks,  and  skills  and 
ensured  that  all  participants  in  the  exercise  would 
receive  useful  training. 

PACIFIC  LINK  2  SYSTEMS 
AFRL  Mesa  Systems 

Constructive  friendly  and  adversary  forces  were 
populated  into  the  scenario  via  computer  generated 
entities  created  from  a  specially  developed, 
unclassified  load  of  AFRL’s  Experimental  Common 
Immersive  Theater  Environment  (XCITE)  software. 
XCITE  is  a  research  and  development  continuation  of 
the  Next  Generation  Threat  System  (NGTS),  the 
software  used  during  PL1.  Similar  to  NGTS,  XCITE 
uses  physics-based  models  to  generate  aircraft, 
weapons,  radar,  and  electronic  warfare  models  from 
parametric  data,  but  unlike  NGTS,  XCITE  has 
undergone  extensive  development  for  increases  in 
capability  and  user  functionality. 

Since  XCITE  is  US  Government  owned  source  code, 
required  modifications  to  the  constructive  environment 
were  easily  implemented  for  PL2.  One  major 
improvement  to  XCITE  was  the  development  of  new 
Threat  Instructor  Operator  Station  (10 S)  software.  The 
Threat  IO S  gives  the  operator  far  more  control  in 
directing  the  constructive  forces  while  adhering  to 
Windows  standards  for  intuitive  use.  Operators  can 


easily  drop  single-,  2-,  3-,  or  4-ship  formations  of 
aircraft  at  specified  speeds,  altitudes  and  headings. 
These  aircraft  formations  can  also  be  saved  into 
Favorites  for  later  quick  retrieval  at  any  time.  The  new 
entities  are  automatically  configured  for  very  realistic 
formation  flying  and  leader  /  wing  command  and 
control  structure.  Operators  can  direct  constructive 
aircraft  to  change  headings,  fly  routes,  form  up  with 
both  XCITE  and  non-XCITE  players,  and  perform  a 
variety  of  tactically-based  combat  maneuvers.  For 
example,  the  operator  can  command  aircraft  orbiting 
over  a  holding  point  to  fly  a  route  through  a  hostile 
environment.  The  lead  aircraft,  after  scanning  a  hostile 
entity,  can  autonomously  direct  its  wingman  to  attack 
the  target  or  engage  the  aircraft  itself.  The  autonomous 
aircraft  will  turn  to  maintain  RADAR  lock  and  fire  the 
most  appropriate  munitions  when  they  are  within 
range.  If  they  are  targeted  themselves,  the  aircraft  will 
automatically  begin  jamming  the  enemy’s 
transmissions.  The  XCITE  models  driving  the 
weapons  engagements,  jamming,  and  emitter  modes 
are  validated  and  based  on  actual  avionics  simulations. 
At  any  point  during  the  scenario,  the  operator  can  take 
total  control  of  the  entities  to  attack  other  targets  or 
perform  a  variety  of  defensive  or  navigational 
maneuvers. 

Due  to  network  and  security  constraints,  an 
unclassified  database  was  used  to  drive  the  XCITE 
emitter,  aero,  and  weapons  models.  During  scenario 
development,  it  was  noted  that  certain  threat  models 
were  unavailable  in  the  unclassified  XCITE  database. 
AFRL  engineers  were  able  to  rapidly  integrate,  on  the 
order  of  days,  over  20  weapons  including  air-to-air  and 
surface-to-air  missiles  and  multiple  aircraft  platforms. 
This  vastly  increased  scenario  fidelity,  depth,  and 
interactivity  for  both  the  manned  F-16  simulators  and 
the  C2  players.  Two  multi-screen  XCITE  consoles 
were  constructed  to  manage  scenario  development, 
White  force  control,  and  act  as  manned  Blue  force 
players.  These  stations  were  integrated  with  ASTi 
compatible  DISVOX  software  driven  radios  for  voice 
communications. 

Warfighter-in-the-Loop  (virtual)  simulators  in  Mesa 
consisted  of  two  F-16  Deployable  Tactics  Trainers 
(DTTs)  plus  two  desktop  LiteFlite  F-16  simulators 
(SDS  International).  DTTs  are  high-fidelity  F-16 
simulators  developed  at  AFRL  based  on  full-fidelity 
Multi-Task  Trainers  (Carr  &  Hernandez,  2005).  The 
DTTs  operate  using  software  converted  from  the  F- 
16  Block  Operational  Flight  Program  and  were 
equipped  with  three  Apple  30-inch  High  Definition 
Cinima  Displays  with  AAcuity  Image  Generators  (SDS 
International)  providing  90°  x  12°  out-the-window 
display  together  with  a  Head-Up  Display. 
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Figure  1.  An  AFRL  Mesa  DTT  Cockpit  Engages 
and  Enemy  Fighter 


The  desktop  F-16  simulators  consisted  of  PCs  running 
SDS  International  LiteFlite  software  and  AAcuity 
image  generators.  This  flight  code  and  image 
generator  runs  on  standard  personal  computers.  One 
1 9  inch  computer  screen  was  used 


Figure  2.  An  AFRL  Mesa  F-16  Pilot  Operating  a 
Desktop  Simulator  in  PL2 


for  the  visual  display  although  pilots  could  control  the 
viewpoint  from  forward  view  with  superimposed  head- 
up  display  to  left,  right,  rear,  or  stealth  view.  Virtual 
and  constructive  simulators  in  PL2  used  the  DIS 
protocol  (IEEE  1278.1a)  external  interface. 

Exercise  recording  was  conducted  with  AFRL’s 
Distributed  Control  Station  (DCS)  software.  DCS  was 
also  used  to  initialize  and  control  the  manned  DTT  and 
desktop  simulators  as  well  as  providing  DIS  log  files 
for  mission  debrief  assessment,  and  engineering 
analysis.  Communications  between  exercise  control, 
SIMOPs,  and  participants  at  the  two  sites  were 
provided  via  a  DIS  voice  network  composed  of 
software-based  devices. 


DSTO  Melbourne  and  Common  Systems 

Several  key  components  of  the  Air  Defence  Ground 
Environment  Simulator  (ADGESIM),  a  suite  of 
simulation  applications  developed  by  DSTO  for  ABM 
training  and  commercially  supported  by  Ytek  Pty  Ltd 
(Melbourne,  Australia),  were  utilised  during  PL2.  The 
ADGESIM  Pilot  Simulation  Interface  (PSI)  provides 
scalable,  real-time  control  for  a  variety  of  entity  models 
via  a  customised  version  of  the  COTS  product  ‘VR- 
Forces’  (MAK  Technologies,  Cambridge,  MA).  The 
PSI  was  used  by  a  RAAF 


Figure  3.  DSTO's  Simulated  C3  Environment 


Simulation  Operator  (SIMOP)  to  provide  minor  Red 
force  elements,  Blue  force  High  Value  Assets  (HVAs), 
and  neutral  entities.  The  sensor  modeling  and  gateway 
application,  known  as  SensorLink,  was  used  to  model 
ground-based,  airborne,  fixed  and  mobile  sensor 
platforms.  Sensors  attached  to  any  given  entity  within 
the  simulation  and  detection  volumes  could  be  culled 
by  Digital  Terrain  Elevation  Data  for  a  specific  region 
or  limited  by  radar  horizon.  SensorLink  translates 
detections  of  DIS  entities  and  outputs  a  real-world 
message  format  of  choice  for  input  to  a  C2  system. 

The  RAAF  ABM  team  required  relatively  little  training 
for  PL2  because  the  operator  systems  utilised  were 
identical  or  very  similar  to  those  available  in  the 
operational  environment.  A  Solipsys  Tactical  Display 
Framework  (TDF)  prviding  customisable  HMI  was  the 
primary  operator  interface  for  track  and  sensor  data. 
The  TDF  is  connected  to  the  Solipsys  Multiple  Source 
Correlation  and  Tracker  (MSCT)  server  which  receives 
a  range  of  sensor  inputs. 

The  TDF  does  not  adequately  support  the  shared 
situational  awareness  requirements  of  RAAF  ABM 
teams  so  an  ad  hoc  tool  known  as  the  ‘toteboard’  was 
developed  to  support  real-time  asset  management. 
This  macro  intensive,  shared  Excel  spreadsheet  lists  all 
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the  air  assets,  and  corresponding  IFF  codes,  assigned  to 
a  C2  agency.  For  each  asset  the  current  alert  state,  fuel 
status  and  weapon  status  may  be  entered  either  directly 
or  via  macro.  Each  team  member  displays  the 
toteboard  adjacent  to  the  TDF  and  enters  information 
on  aircraft  within  their  area  of  responsibility.  Updates 
to  an  individual  view  of  the  spreadsheet  occur  on  a 
push-pull  basis,  and  is  stored  centrally. 

A  further  component  of  ADGESIM  is  the  Tactical 
After  Action  Review  for  DIS  (TAARDIS)  application, 
a  tool  is  capable  of  capturing  DIS  ground  truth  and 
operator  ‘perceived’  truth  (via  console  screen  capture) 
and  replaying  both  in  a  synchronous  manner  to  any 
DIS  viewer  and  Windows  Media  Player  respectively. 
Performance  data  were  collected  via  the  Mentor 
software  system  (Calytrix  Technologies,  Perth, 
Western  Australia).  Mentor  is  a  performance 
assessment  package  that  provides  for  exercise 
definition,  measurement  and  the  review  of  outcomes. 
Data  collection  is  facilitated  by  the  Data  Entry  Tool 
(DET)  which  is  loaded  with  an  assessment  form 
dynamically  generated  by  the  main  Mentor  application. 

Polycom  teleconferencing  systems  (Pleasanton, 
California,  USA)  were  used  at  both  sites  for  video 
conferencing  and  to  capture  video  of  the  ABM  team. 
Smartboards  (Smart  Technologies  Inc.,  Calgary, 
Canada)  were  utilised  for  displaying  and  enhancing 
(via  electronic  pens)  session  briefings  and  AARs. 
Ultra VNC,  an  open  source  remote  desktop  viewer,  was 
used  to  share  briefings  data  recordings  between 
distributed  participants.  Due  to  the  bandwidth  and 
latency  constraints  of  the  WAN  several  combinations 
of  quality  and  compression  settings  were  trialed  before 
an  acceptable  level  of  performance  was  achieved. 
Wireshark,  an  open  source  application,  was  used  to 
conduct  network  traffic  analysis. 

Network 

Connectivity  was  established  between  DSTO 
Melbourne  and  AFRL  Mesa  through  an  unclassified 
network  of  three  separately  administered  segments:  the 
DSTO  internal  network  between  Melbourne  and 
Adelaide;  the  Australian  Academic  Research  Net 
(AARNet)  network,  between  Adelaide  and  Seattle;  and 
the  US  Defense  Research  and  Engineering  Net 
(DREN),  connecting  Seattle  with  Mesa. 


PARTICIPANTS 

Exercise  participants  consisted  of  an  ABM  team  (a 
weapons  director  (WD)  plus  three  fighter  controllers) 
and  an  ABM  assessor/instructor  at  DSTO  Melbourne, 


with  four  F-16  pilots  at  AFRL  Mesa.  The  F-16  pilot 
training  audience  was  represented  by  Subject  Matter 
Experts,  all  current  or  former  experienced  F-16  pilots 
familiar  with  distributed  mission  training  exercises. 
AFRL  Mesa  SIMOPs  controlled  constructive  Red 
Threat  and  Blue  interactive  forces  from  the  dual 
XCITE  consoles.  Senior  Exercise  Controllers 
(EXCON)  were  placed  in  Melbourne  and  Mesa.  Both 
sides  had  engineering  staffs,  exercise 
managers/observers,  and  data  collectors. 

The  RAAF  ABM  team  had  a  mix  of  experience:  the 
WD  was  very  experienced  (10+  years,  500+  controlling 
hours;  B  CAT  Fighter  Controller  and  Weapons 
Director);  one  of  the  controllers  had  medium 
experience  (2  years,  200+  controlling  hours;  C  CAT 
Fighter  Controller);  and  the  remaining  team  members 
had  limited  experience  (1  year,  50  controlling  hours;  D 
CAT  Fighter  Controller).  This  represented  a  fairly 
average  team.  The  team  had  not  worked  together 
previously  as  a  team,  although  individual  controllers 
had  had  some  experience  of  working  together.  The 
senior  members  of  the  team  had  previous  experience 
working  live,  large  force  coalition  training  exercises 
(e.g.,  Pitch  Black,  Talisman  Saber),  but  none  of  the 
team  had  participated  in  virtual  forms  of  such 
exercises. 


SCENARIO  DEVELOPMENT 

PL2  provided  an  opportunity  to  develop  and  trial  a 
structured  method  for  designing  training  scenarios  for 
RAAF  ABM  teams  which  drew  on  ideas  from  the 
USAF  AW  ACS  MECs  which  describe  the  knowledge, 
skills  and  experiences  required  for  successful  mission 
completion  (Colegrove  &  Alliger,  2002).  In  addition, 
research  on  DMT  and  team  training  were  linked  into 
an  Australian  task  framework:  the  Australian  Joint 
Essential  Task  list  (ASJETS;  McCarthy  et  al,  2003). 

An  event-based  approach  to  training  design  (e.g., 
Fowlkes,  Dwyer,  Oser,  and  Salas,  1998),  which 
focuses  on  the  deliberate  presentation  to  trainees  of 
opportunities  to  demonstrate  and  learn  specific  classes 
of  competencies  through  the  introduction  of  specific 
types  of  scenario  events,  was  developed  to  guide  the 
PL2  scenario  definition.  Each  event  is  designed  to 
create  the  requirement  to  act  in  a  way  which  places 
demands  on  an  identified  set  of  competencies. 
Observations  of  behaviours  related  to  these 
competencies  are  made  and  the  successful  (or 
otherwise)  demonstration  of  these  behaviours 
contributes  to  training  outcomes.  The  scenario  design 
effort  focused  on  the  definition  of  actual  events  to  be 
included  in  the  PL2  scenarios  which  were  built  in 
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XCITE  and  validated  to  ensure  accurate 
implementation  meeting  scenario  objectives. 

SCENARIO  IMPLEMENTATION 

The  exercise  was  conducted  over  four  days,  with  the 
background  being  protection  of  a  UN  humanitarian 
relief  operation  by  coalition  air  assets.  The  focus  was 
on  Defensive  Counter  Air  (DCA)  with  escalating 
tensions  and  engagements  leading  to  an  Offensive 
Counter  Air  (OCA)  mission  on  Day  3.  Day  4  returned 
to  a  predominantly  DCA  focus.  Each  session  followed 
a  general  exercise  scenario  of  escalating  hostility  and 
increasing  task  complexity. 

AFRL  Mesa 

XCITE  provided  as  many  as  five  4-ships  of  Blue  F- 
1 6  aircraft,  multiple  formations  of  MiG  fighter  aircraft, 
early  warning  radars,  surface-to-air  missiles,  and 
ground  targets.  Scenarios  included  ATO  holding  CAP 
points,  air  refueling  tracks,  and  routes  were  built  and 
saved  with  the  XCITE  Threat  IOS  prior  to  the  start  of 
the  exercise.  A  subject  matter  expert  operated  the 
radios  during  runtime  for  the  constructive  forces  and 
controlled  them  via  the  Threat  IOS.  During  periods  of 
high  activity,  another  operator  would  help  control  the 
entities  with  a  second  Threat  IOS  connected  to  the 
same  XCITE  server.  Since  as  many  as  20  constructive 
forces  were  operating  within  the  scenario  at  the  same 
time,  a  key  factor  in  keeping  them  all  flying  properly 
with  only  two  operators  was  XCITE’ s  ability  for 
autonomous  control.  Rather  than  micromanage  the 
flight  of  all  constructive  forces,  the  IOS  operators  were 
able  to  issue  simple  commands  to  the  aircraft  such  as 
“Form  Up”  to  the  tanker,  “Follow  Route”  along  a 
designated  route,  or  “Attack”  a  specific  target  with  a 
specific  munition.  The  XCITE  entities  would  then 
handle  all  necessary  flight  controls  until  their  tasks 
were  completed  or  the  operator  commanded  them 
differently.  If  no  changes  were  made  to  a  group  of 
constructive  aircraft’s  tasks,  then  there  was  little-to-no 
additional  burden  on  the  operators  to  continue  flying 
those  aircraft. 

To  keep  the  pilots  of  the  virtual  simulators  engaged  as 
much  as  possible  without  disrupting  the  scenario  play, 
the  cockpits  were  periodically  hot-swapped  with  the 
constructive  forces.  For  example,  one  constructive 
formation  of  F- 16s  may  have  been  formed  up  with  the 
refueling  tanker,  a  second  formation  may  have  been 
waiting  at  the  base  for  launch,  and  a  third  formation 
may  have  been  following  a  route  to  a  holding  point  just 


Figure  4.  An  AFRL  Mesa  SME  Operating  the 
XCITE  Constructive  Forces  in  PL2 


outside  the  hostile  environment.  As  the  pilots  in  the 
virtual  cockpits  left  the  hostile  area,  constructive 
entities  were  dropped  in  their  place.  Next,  the  virtual 
cockpits  were  repositioned  onto  the  route  to  take  the 
place  of  the  constructive  entities  there.  These 
interactions,  requiring  substantial  amounts  of 
coordination  between  the  pilots,  cockpit  operators,  and 
XCITE  operators,  were  performed  several  times 
throughout  the  exercise  and  ultimately  presented  a 
better,  more  active  training  environment  for  the  pilots. 
Experiences  from  the  PL2  Exercise  identified  25 
Threat  IOS  and  XCITE  improvements  for  increased 
functionality,  stability,  and  ease  of  use. 

Very  detailed  scenario  events  were  implemented  by  the 
XCITE  software  and  IOS.  Throughout  the  exercise  the 
F-16  pilots  conducted  interactive  composite  force 
tactical  operations  with  constructive 
players  including  lane  and  target  handoffs,  combined 
flight  targeting,  and  integrated  strike  packages. 
Controllers  actively  managed  lane  and  CAP  tasking, 
tanker  flow,  and  CAP  re-set  procedures,  as  well  as 
specific  tactical  engagements.  In  one  unique  event, 
one  of  the  constructive  F-16  players  was  damaged  and 
flew  a  realistic  battle  damage  route  to  include  lost 
communications  supporting  a  controller  learning 
objective. 

DSTO  Melbourne 

Tanker,  UAV,  and  neutral  aircraft  were  constructed  in 
PSI/VR-Forces  at  runtime.  A  single  SIMOP 
manipulated  these  entities  by  directing  them  manually 
or  by  assigning  them  to  a  pre-determined  or  quickly 
constructed  route.  SIMOP  also  provided  voice  acting 
for  the  tanker  aircrew,  UAV  operator,  UN  pilots,  and 
civilian  aircraft  captains.  Some  entities  took  a 
legitimate  path  from  a  known  base  of  operations  (even 
though  this  was  beyond  sensor  coverage)  and  others 
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were  inserted  directly  into  the  area  of  operations  to  test 
the  team’s  reactions  to  pop-up  threats.  During  periods 
of  high  workload  a  member  of  Exercise  Control  would 
accept  responsibility  for  a  subset  of  the  entities  in  terms 
of  their  manoeuvre,  targetting  and  weapons 
employment.  As  a  result  of  PL2,  several  suggestions 
were  made  to  improve  the  manipulation  of  grouped 
entities  within  the  PSI. 

Exercise  Control  also  manipulated  the  state  of  the 
sensor  feeds  during  the  scenario.  Within  the  Sensor- 
Link  package  radars  were  degraded  when  not  expected, 
depending  on  the  desired  training  outcome.  On 
occasion  the  parameters  of  one  sensor  were  modified  to 
reduce  low  level  coverage  around  an  exclusion  zone. 

EXERCISE  OUTCOMES 
Exercise  Evaluation 

Team  performance,  training  outcomes,  and  the  conduct 
of  the  exercise  were  assessed  using  a  range  of  self- 
report  measures  (by  members  of  the  ABM  team  and 
pilots),  measures  of  Team  Objectives  (by  a  qualified 
instructor/assessor  using  the  MENTOR  tool),  and 
exercise  evaluation  measures  (by  all  participants).  Pre¬ 
exercise  the  ABM  team  members  recorded  their 
expectations  of  the  effectiveness  of  DMT  for  ABM 
training  and  their  training  objectives.  'During  each  day 
the  ABM  team  members  rated  their  own  and 
teammates’  workload.  The  US  pilots  also  completed 
measures  of  coalition  team  efficacy,  cohesion,  and 
team  processes  each  day.  Teamwork  measures 
(commitment,  efficacy,  cohesion,  team  process,  and 
workload)  would  provide  a  snapshot  of  the  team  at 
each  stage  of  the  exercise,  and  an  indication  of  the 
extent  to  which  teamwork  (both  within  the  ABM  team 
and  across  the  coalition  team)  was  improving  over  the 
course  of  the  exercise.  This  would  highlight  the  value 
of  DMT  for  ABM  training  and  the  enhancement  of 
coalition  interoperability. 

The  measures  of  Team  Objectives  were  collected 
during  each  mission  by  the  instructor/assessor,  and  a 
stoplight  report  was  generated  using  the  MENTOR  tool 
at  the  end  of  each  mission  as  a  guide  for  the  debrief  of 
the  ABM  team.  Team  behaviours  were  assessed  across 
Mission  Planning  and  Review;  Manage  Information 
Systems;  Control  Airspace;  Conduct  Defensive 
Counter  Air;  Military  Liaison;  Protect  Key  Points  and 
Vital  Assets;  and  Supporting  Competencies.  The  trend 
with  these  measures  across  the  exercise  would 
highlight  the  training  value.  The  exercise  evaluation 
measures  focused  on  the  way  the  exercise  was 
conducted  and  were  filled  out  at  the  end  of  each  day  by 
all  participants  to  capture  what  went  right/wrong. 


These  measures  were  especially  useful  to  highlight 
particular  procedural  and  technical  issues  that 
enhanced/constrained  performance. 

ABM  Training  Reactions 

Overall,  the  ABM  team  found  the  exercise  scenario 
met  their  expectations  and  provided  significant  combat 
mission  training  value,  especially  for  team  coordination 
skills.  These  kinds  of  missions  are  infrequent  due  to 
cost  and  availability  issues,  and  the  simulation  facilities 
required  to  support  such  training  are  not  currently 
available  at  their  home  unit.  They  found  the  scenarios 
were  well  structured,  paced,  and  sequenced  in  a  way 
that  facilitated  their  learning.  They  also  reported  that 
they  would  be  motivated  to  seek  similar  training 
opportunities  in  the  future,  and  would  recommend  the 
training  to  other  controllers.  They  rated  it  a  valuable 
experience  in  coalition  training,  providing  exposure  to 
US  forces  and  doctrine,  and  recommended  that  these 
kinds  of  exercises  should  be  part  of  all  future  coalition 
spin-ups/preparations.  The  ABM  team  indicated  that 
the  exercise  provided  good  training  experiences  for  the 
basic  mechanics  of  tactical  air  battle  management  (e.g., 
detecting,  identifying,  and  tracking  entities;  force 
marshalling;  tanker  management;  maintaining  tracking 
and  safe  control  of  a  number  of  groups;  high  value 
asset  protection;  return  to  force  procedures/lame  duck 
procedures/safe  passage;  working  with  multiple  radio 
control  frequencies).  In  addition,  the  scenarios 
presented  effective  training  experiences  of  non¬ 
standard  formations,  detection  and  tracking  of  high  fast 
flyers,  and  leaker  scenarios.  Pre-exercise,  the  ABM 
team  did  not  expect  complex  scenarios  (e.g.,  tracking 
large  Red  force  groups,  controlling  large  Blue  force 
groups,  working  with  combat  search  and  rescue  assets, 
working  with  complex  ROE  [rules  of  engagement])  to 
be  effectively  trained  in  a  distributed  simulation 
environment.  However,  following  the  exercise  the 
team  indicated  that  the  training  for  these  complex 
mission  elements  was  effective. 

Scenario  Fidelity 

The  scenarios  allowed  team  members  to  effectively 
measure  team  objectives  in  critical  events.  Events  like 
enforcement  of  ADIZ  procedures,  identifying  an 
emergency  (an  aircraft  flying  a  triangle  pattern), 
adhering  to  the  ROE,  and  monitoring  fighter  fuel 
states  of  the  fighters  were  effectively  represented  using 
the  combination  of  virtual  and  constructive  players. 

Coalition  Training  Outcomes 

At  the  end  of  the  exercise  the  Australian  ABM  team 
and  US  pilots  rated  coalition  interoperability  training. 
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In  general,  the  coalition  team  agreed  that  there  was  a 
high  level  of  trust  between  coalition  partners,  that  this 
increased  over  the  course  of  the  exercise,  and  the 
military  doctrine,  experience,  and  training  enabled  the 
coalition  partners  to  work  effectively  together. 
Overall,  there  was  evidence  that  the  exercise  provided 
training  that  enhanced  key  interoperability  factors.  The 
ABM  team  highlighted  several  coalition  challenges  as 
a  result  of  their  exercise  experience,  including:  (i) 
while  language  and  terminology  were  only  slightly 
different,  this  had  an  impact  during  engagements,  (ii) 
there  were  differences  in  doctrine  and  interpretations  of 
ROE  that  were  not  easily  overcome  through  briefing 
sessions,  (iii)  differences  in  standard  operating 
procedures  were  not  a  problem  in  themselves,  but 
knowledge  of  these  differences  meant  that  more 
explicit  communication  was  required,  and  (iv) 
allocation  of  roles  and  resources  at  a  tactical  level 
differed  due  to  the  different  capabilities  of  the 
participants  in  the  coalition  environment. 


EXERCISE  EVALUATION 
Network  Infrastructure 

The  network  that  was  used  for  PL2  performed 
satisfactorily  in  supporting  distributed  briefing, 
debriefing,  and  mission  execution.  While  this  network 
route  was  longer  than  optimal  due  to  the  availability  of 
peering  points,  long  term  measurement  of  the  network 
using  ping  revealed  a  steady  round  trip  time  of  238ms, 
which  proved  acceptable  for  this  kind  of  training 
exercise.  The  DREN  and  AARNet  components  of  the 
network  could  easily  support  bandwidth  of  lOOMbit/s, 
but  a  limitation  of  1  Mbit/s  incurred  in  the  internal 
DSTO  network  constrained  the  resolution  of  electronic 
materials  that  could  be  shared  during  mission  debriefs. 
Furthermore,  the  video  conference  system  and  remote 
desktop  software  demonstrated  different  effective 
latencies.  This  restricted  the  rate  at  which  briefs  could 
progress,  as  the  appearance  of  complex  images  or 
replays  at  the  remote  end  was  delayed  and,  therefore, 
poorly  synchronised  with  narrative  of  the  speaker. 
This  was  overcome  by  confirming  that  the  desired  data 
had  been  received  at  the  remote  end  before 
commencing  a  related  discussion. 

Increasing  the  bandwidth  available  for  exercises  would 
allow  greater  flexibility  in  the  use  of  collaboration 
technologies  and  content.  This  would  also  have  an 
effect  on  latency,  as  insufficient  bandwidth  can  result 
in  queuing.  Future  activities  should  seek  to  determine 
the  level  at  which  there  is  no  significant  network 
degradation.  While  latency  may  remain  an  issue  for 
some  systems,  this  could  be  countered  procedurally  or, 


by  utilising  systems  that  exhibit  the  least  latency.  For 
example,  the  distribution  of  replay  data  may  be  more 
robust  than  the  sharing  of  display  data. 

Sub-system  reliability 

Several  of  the  applications  that  were  used  exhibited 
some  instability;  however,  as  most  were  recoverable, 
this  did  not  significantly  degrade  the  exercise.  In  one 
instance  a  SROCCS  device  crashed  and  impacted  the 
check-in  of  fighter  aircraft,  and  in  another,  XCITE 
crashed  to  abruptly  end  the  OCA  scenario.  The 
reliability  of  simulation  systems  has  improved  since 
PL1,  yet  software  problems  remain.  This  is  likely  to 
continue  as  systems  are  developed  and  may  only  be 
mitigated  by  regular,  structured  distributed  simulation 
testing. 

Performance  Measurement 

Performance  measurement  and  feedback  in  PL2 
centred  on  the  use  of  the  Mentor  training  management 
system.  The  Mentor  system  provided  the  facility  to 
manage  objectives,  to  undertake  performance 
assessment  during  missions,  and  to  provide  feedback 
during  after  action  review.  Mentor  received  positive 
reviews  from  the  ABM  team.  In  particular,  they  found 
the  ability  to  view  performance  on  similar  objectives 
over  time  valuable,  as  it  was  informative  about  the 
etiology  of  any  marginal  or  poor  performance  that  was 
observed.  The  Supporting  Competencies  section  of 
the  Mentor  assessment  form  was  not  used  effectively 
by  the  team  assessor  and  due  to  assessor  workload  and 
was  often  completed  at  the  end  of  a  session.  If 
assessing  teamwork  is  a  high  priority,  a  separate 
assessor  should  focus  on  this  dimension. 

Scenario  Design 

The  process  event-based  proved  relatively 
straightforward  to  implement  and  was  found  to  be  quite 
useful  in  guiding  researchers  and  SMEs  during  the 
development  of  the  exercise  by  linking  relevant  team 
tasks  to  scenario  design.  The  approach  was  grounded 
in  previous  research  on  team  training,  AW  ACS  MECs, 
and  the  event  based  approach  to  training  and  had  the 
advantages  of  a  relatively  rapid  development  cycle  and 
relatively  low  resource  requirements. 

In  general  the  participants  reported  that  the  scenarios 
were  well  structured  and  engaging.  However,  they 
identified  some  aspects  of  the  scenarios  that  could  have 
been  improved.  For  example,  a  greater  appreciation  of 
Red  Force  activities  could  have  been  afforded  through 
the  provision  of  intelligence  reports,  realistic  indicators 
and  warnings.  This  could  have  allowed  the  ABM  team 
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to  manage  the  application  of  resources  over  the 
duration  of  the  operation  more  appropriately. 

Exercise  management 

The  coordination  of  exercise  control  between 
distributed  sites  was  problematic.  It  required  detailed 
scenario  guides  to  be  shared  and  interpreted  in 
advance,  as  well  as  runtime  co-ordination  to  ensure 
good  situation  awareness  of  the  implementation  of 
events.  This  was  complicated  by  the  dynamic  nature  of 
some  scenarios  which,  despite  the  planning  of  trigger 
events,  did  not  eventuate  as  expected.  This  required  ad 
hoc  re-planning  and  significant  communication 
between  the  sites  to  ensure  effective  coordination. 
Scenario  preparation  also  required  input  from  the  ABM 
team  to  adjust  their  plan  prior  to  the  next  session. 
Sometimes  this  process  was  abbreviated  due  to  the  late 
arrival  of  the  planning  materials.  This  occurred  for  the 
OCA  scenario,  and  as  the  ABM  team  was  also  less 
familiar  with  this  kind  of  mission,  there  was  a 
perceived  negative  impact  upon  participant 
performance. 

The  roles  and  procedures  for  exercise  management 
need  to  be  documented  and  agreed  well  in  advance  of 
the  exercise.  These  should  include  commitments  to 
supply  specific  sets  of  information  within  a  specific 
timeframe.  If  the  overall  scenario  is  to  be  fixed,  rather 
than  modified  over  time  due  to  participant  behaviour, 
then  the  participants  should  be  permitted  to  perform 
their  planning  processes  in  advance.  If  a  dynamic 
scenario  is  desired  then  adequate  participant  planning 
time  should  be  allocated  between  sessions.  If  the 
participants  are  to  encounter  new  kinds  of  missions 
additional  preparation  may  be  required  to  ensure  valid 
outcomes. 

Mission  execution 

There  were  several  distributed  team  co-ordination 
issues  that  degraded  collective  mission  effectiveness. 
These  typically  involved  misunderstandings  between 
the  various  Blue  force  participants  and  White  force  role 
players  regarding  the  procedures  to  be  implemented. 
These  problems  resulted  in  outcomes  such  as  delays  to 
time  critical  actions,  breaches  of  ROE,  fratricide  and 
reduced  trust  in  other  participants.  In  almost  all  cases 
these  were  overcome  by  providing  greater  detail  and 
clarification  in  briefings.  There  were  also  some 
differences  in  communication  terminology  (RAAF 
variations  on  the  US  3-1  standard)  that  caused  minor 
confusion,  but  did  not  appear  to  significantly  affect  the 
outcome. 

Communication  bottlenecks  were  encountered  on  a 
number  of  occasions  with  respect  to  those  Blue  force 


members  controlling  multiple  constructive  4-ship 
formations.  At  times  only  one  SME/SIMOP  was 
available  to  provide  communication  interaction  and 
tactical  decision  making  for  up  to  three  formations. 
The  level  of  automation  provided  by  the  XCITE  entity 
generator  did  not  alleviate  the  requirement  for  timely 
and  effective  responses  to  ABM  requests.  There  were 
many  instances  of  missed  calls  and  significantly  slow 
responses  during  high  tempo/workload  events.  The 
OCA  scenario  differed  to  other  sessions  in  that  it 
required  SIMOPs  otherwise  responsible  for  Blue  forces 
to  also  provide  Red  forces.  As  this  type  of  mission 
required  concentrated  and  coordinated  actions, 
communication  delays  degraded  the  ability  of  the  ABM 
team  to  counter  threats,  which  diminished  fidelity  and 
therefore  training  value. 

Several  of  the  above  co-ordination  issues  are  typical 
outcomes  of  team  training  events,  and  the  rectification 
of  these  issues  correlates  with  improved  performance. 
However,  where  technical  or  procedural  constraints 
reduce  task  fidelity  below  an  acceptable  level,  negative 
learning  can  result.  Whilst  addressing  these  issues  may 
be  straight-forward,  the  resource  implications  may 
affect  the  feasibility  of  conducting  these  activities. 
Simulation  operator  and  other  White  force  manning 
must  be  comensurate  with  the  desired  scenario 
intensity  and  complexity.  If  the  rate  of  effort  is 
unsustainable,  technologies  to  reduce  manning 
requirements  should  be  integrated  with  simulation 
tools.  These  may  include  voice  recognition,  machine 
speech,  and  intelligent  teamed  agents. 

Simulation  Fidelity 

While  the  simulation  environment  was  generally 
sufficient  to  achieve  the  desired  training  objectives,  a 
lack  of  fidelity  in  some  aspects  and/or  components  had 
a  negative  impact  on  training  effectiveness.  Some  of 
the  Blue  force  simulators  did  not  have  the  ability  to 
readily  set  the  IFF  modes  and  codes  required  by  the 
ABM  team  to  monitor  friendly  aircraft,  which  created 
confusion.  For  example,  when  two  such  aircraft  left 
the  airspace  and  returned  they  were  mis-identified  as 
hostile.  The  Melbourne  SIMOP  was  not  advised  to 
avoid  constructing  specific  entity  types  in  PSI/VR- 
Forces,  the  enumerations  of  which  had  not  been 
verified  against  Mesa  systems.  Unfortunately  this 
occurred  and  a  UN  Medivac  Fearjet  was  portrayed  as 
F-15  within  the  virtual  cockpits.  Similarly,  terrain  was 
not  represented  consistently  between  simulation 
systems.  The  Mesa  virtual  cockpits  modeled  and 
displayed  terrain  whilst  the  DSTO  constructive 
simulator  did  honour  terrain.  In  one  instance  a  Red 
aircraft  was  able  to  fly  within  the  terrain  and  elude  the 
Blue  fighters.  The  fidelity  limitations  mentioned  above 
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were  largely  procedural;  however,  there  were  others 
attributable  to  the  nature  of  the  systems. 

Th  restriction  of  operating  within  an  unclassified 
environment  prevented  the  use  of  the  most  realistic 
weapon  and  sensor  models.  The  DSTO  constructive 
simulators  did  not  support  an  external  electronic 
warfare  (EW)  environment  interface.  Therefore  the 
Red  entities  created  within  did  not  stimulate  the  radar 
warning  receiver  (RWR)  on  any  of  the  F- 16  simulators. 
This  made  aspects  of  identifying  hostile  intent  in 
accordance  with  ROE  and  threat  avoidance  difficult  or 
impossible  to  perform,  and  resulted  in  negative 
learning.  Additionally,  the  limited  number  of  Blue 
force  SIMOPs  meant  that  there  were  times  “when  you 
knew  that  your  sim  pilot  was  busy  talking  with  another 
controller,  so  key  calls  on  the  radio  were  not  made”. 
This  may  be  an  unintended  example  of  team  workload 
balancing  behaviour;  however,  it  degraded  the  ability 
to  achieve  the  core  training  objectives. 


CONCLUSIONS 

The  key  objectives  of  Exercise  PL2  were  to:  (i) 
develop  systems  which  allow  small  White  force  to 
support  ABM  training  (ii)  continue  development  of 
AUS-US  distributed  simulation  infrastructure,  (iii) 
continue  development  of  performance  measurement 
and  feedback  techniques  for  coalition  DMT,  (iv) 
develop  a  structured  method  for  designing  ABM  team 
training  scenarios,  and  (v)  investigate  interoperability 
issues  associated  with  air  battle  management  and  air 
combat  missions  for  coalition  operations. 

The  scenario  network  that  was  used  for  PL2  performed 
well  in  supporting  distributed  briefing,  debriefing,  and 
mission  execution.  This  demonstrates  that  coalition 
mission  training  of  this  kind  is  feasible  as  an  adjunct  to 
regular  training  for  ABM  teams  and  fighter  aircrew. 
Importantly,  expert  assessment  of  the  ABM  team 
revealed  a  performance  increase  over  the  course  of  the 
exercise.  This  demonstrates  that  there  is  indeed  a 
benefit  to  undertaking  such  training.  With  the 
exception  of  a  small  number  of  systems  and  procedural 
issues,  participant  reactions  to  the  exercise  were 
overwhelmingly  positive.  This  indicates  that  training 
of  this  kind  is  viewed  as  beneficial  and  is  likely  to 
garner  a  great  deal  of  acceptance  as  it  gains  exposure 
within  the  Australian  operational  community,  as  it  has 
in  the  US.  These  evaluations  provide  a  solid 
foundation  for  advancing  the  development  of  tools  and 
techniques  for  conducting  coalition  DMT  between 
Australia  and  the  US  in  the  air  combat  domain.  Future 
plans  for  the  Pacific  Link  series  of  exercises  involve 
building  upon  the  infrastructure  and  expertise  gained 


through  the  conduct  of  Exercises  1  and  2.  In  particular, 
it  is  anticipated  that  the  development  of  the  Aerospace 
Battlelab  at  the  DSTO  Melbourne  site  will  enable 
future  exercises  to  incorporate  classified  platforms, 
weapons,  and  sensor  modeling  for  enhanced  realism. 
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